System and method for single channel whole cell segmentation

ABSTRACT

The present disclosure relates to a computer-implemented system and its associated method for single channel whole cell segmentation of a sample image of a biological sample. The biological sample may be stained with one or more non-nuclear cell marker stains, and the system and the method are configured to transform the sample image of the biological sample stained with the one or more non-nuclear cell marker stains into a segmented image having one or more cells with delineated nuclei and cytoplasm regions.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to whole cellsegmentation of an image of a biological sample and, more particularly,to single channel whole cell segmentation of an image of a biologicalsample stained with one or more non-nuclear cell marker stains.

BACKGROUND

A cell is a basic structural, functional and biological unit in allliving organisms. In addition, the cell contains several sub-cellularcompartments (e.g. nucleus, cytoplasm and membrane) and organelles (e.g.mitochondria). The ability to image, segment and study the cell iscentral to research and clinical studies. One example is research tounderstand cellular dynamics in normal and pathological conditions.Another example is drug discovery where it is important to measure theeffect of different drug treatment conditions on the cells. Anotherexample is live cell imaging for studying cellular dynamics of livingcells using a time-lapse microscopy. The recent advancement inhigh-resolution fluorescent microscopy has paved the way for detailedvisualization of the cells and their subcellular structures. Theadvancement in microscopy has been accompanied by the advancement ofcomputing capabilities as well as the development of techniques incomputer vision and image processing for image segmentation, whichpermitted accurate and high-throughput cell analysis.

Cell segmentation, and especially whole cell segmentation, has been thefocus of many research over the last few decades. The term segmentation,as used herein, refers to the identification of boundaries of biologicalunits, such as cells. For example, in a whole cell segmentation,boundaries of cells and sub-subcellular compartments of the cell, suchas a nucleus, cytoplasm and/or membrane, are delineated within a sampleimage. By the whole cell segmentation process, the sample image istransformed into a segmented image with delineated regions. The sampleimage may be obtained using a microscope, for example a fluorescencemicroscope. Achieving accurate segmentation can often be challenging dueto the complexity of the sample images and the high density of thetissue structures which have no obvious boundaries. In certain analysistechniques, segmentation is used to identify regions and/or units forbiomarker quantification and feature extraction (e.g. morphologicalfeatures and phenotyping). For example, a cell analysis workflow mayinvolve cell segmentation, cell level quantification and data analysis.The cell segmentation step may use multiple structural markers tosegment different subcellular compartments and then to delineate cellborders in a whole cell segmentation. In the second step, each biomarkeris quantified at both the cellular level (e.g. mean or total intensitiesfor each cell) and the subcellular level. Then, these cell levelmeasurements are usually aggregated at the image or subject level at thebeginning of the data analysis stage. While whole cell segmentationfacilitates performing detailed cell level quantification, existing cellanalysis workflows have certain drawbacks. For example, errors from thecell segmentation step may result in quantification errors. In addition,detailed cell segmentation and cell level quantification of an image maybe time consuming where the processing time may vary between imagesdepending on tissue characteristics and the number of cells in theimage. In addition, manual review of cell segmentation, which is alsotime-consuming, is often required.

Several techniques for cell segmentation are commonly used. Existingtechniques for whole cell segmentation often rely on two or morechannels of an instrument (for example, a microscope) for performingsegmentation and are referred to herein as “two channel whole cellsegmentation” or “multi-channel whole cell segmentation”. For example,many of the existing techniques rely on the use of both a nuclear cellmarker that specifically stains nucleus in a first channel and one ormore non-nuclear cell markers in a second channel different from thefirst channel for whole cell segmentation. In contrast to the nuclearcell markers, non-nuclear cell markers refer to the cell markers that donot specifically stains nucleus, for example, the membrane and/orcytoplasmic markers. Accordingly, such two-channel or multi-channelwhole cell segmentation techniques require two or more channels with thefirst channel reserved for the nuclear cell marker and the second and/orfurther channels for non-nuclear cell markers.

However, given the limited number of channels of most microscopes, it isvery often desirable to use only a single channel for the segmentationso that the rest of the channels may be available for other analyticalbiomarkers used to study different biological phenomena. Furthermore,there is often a desire, especially in live cell imaging, to performwhole cell segmentation using a single channel of an instrument such asa microscope, referred to herein as a “single channel whole cellsegmentation”, and to avoid the use of nuclear cell markers whichspecifically stain nuclei (for example, DAPI(4′,6-Diamidino-2-Phenylindole)) due to the toxic effects the nuclearcell markers have on the cells and the changes in morphology that mayarise from their use.

Some existing single channel cell segmentation techniques rely on imagefeatures such as intensity and/or morphology. For example, a watershedtransform is an image processing technique that has been used forseparating touching/overlapping cells or nuclei and for segmentingimages of cells. With the watershed transform, a sample image may bemodeled as a three-dimensional topological surface, where values ofpixels (e.g. brightness or grey level) in the image representgeographical heights. Other image processing techniques that have beenused for segmenting images of cells include morphological-basedtechniques, for example, blob-based detection which assume a blob-likeshape for the cell or nucleus. Active contours models and/or snakesalgorithms have also been used. These existing techniques do not,however, permit accurate whole cell segmentation due to the variationsassociated with cells.

For example, due to variations in the histology of different tissuetypes, segmentations may not produce an accurate segmentation withoutsignificant adaptation and optimization for specific tissue typeapplications. It has been noted that a segmentation technique may causethe images to be over-segmented (e.g. what appears as a single cell mayactually be only a portion of a cell) or under-segmented (e.g. whatappears as a single cell may actually be several different cells incombination). Furthermore, suitable segmentation parameters for oneregion of the image may not work well in other regions of the sameimage. Therefore, existing techniques may not be robust enough forsegmentation of large numbers of cells having many morphologicalvariations. In addition, cells are often stained with different markersand imaged under different magnifications, which could lead to a highvariability in cell shape and appearance, thus leading to poorsegmentation results.

More recently, deep learning based techniques have gained a significantinterest in the biomedical image analysis domain. These machine learningtechniques (e.g. pixel classification) have also been applied to cellsegmentation. For example, a deep learning model was used to identifycells of different classes from three channels. However, no actualsegmentation of the cell boundary was performed.

Therefore, it is highly desirable to develop an improved system andmethod for performing single channel whole cell segmentation of imagesof biological samples, more particularly, single channel whole cellsegmentation of images of biological samples stained with non-nuclearcell markers.

SUMMARY

Certain embodiments commensurate in scope with the originally claimedsubject matter are summarized below. These embodiments are not intendedto limit the scope of the claimed subject matter, but rather theseembodiments are intended only to provide a brief summary of possibleembodiments. Indeed, the disclosure may encompass a variety of formsthat may be similar to or different from the embodiments set forthbelow.

In one embodiment, a computer-implemented method is provided fortransforming a sample image of a biological sample stained with one ormore non-nuclear cell marker stains into a segmented image comprisingone or more cells having delineated nuclei and cytoplasm regions. Themethod includes: (a) providing a computer system having: a processorconfigured to execute instructions; and a memory or a storage deviceoperatively coupled to the processor, wherein at least one of the memoryand the storage device is configured to store: a model generated fromtraining data comprising a plurality of training images of biologicalsamples, the training images comprising regions identified as at leastone of nuclei, cells and background; the sample image of the biologicalsample stained with the one or more non-nuclear cell marker stains; andprocessor-executable instructions, (b) accessing, in the memory or thestorage device, the model and the sample image of the biological samplestained with the one or more non-nuclear cell marker stains; (c)generating, by applying the model to the sample image, a nucleiprobability map comprising predicted nuclei regions; and a cellprobability map comprising predicted cell regions; (d) extracting, bythe processor, a binary nuclear mask from the nuclei probability map;(e) extracting, by the processor, a nuclei seeds map from the binarynuclear mask, the nuclei seeds map comprising extracted individualnuclei seeds separated by delineated nuclei regions; (f) applying, bythe processor, the extracted nuclei seeds to the sample image; and (g)transforming the sample image into the segmented image comprising one ormore cells having delineated nuclei and cytoplasm regions.

In another embodiment, a computer system for transforming a sample imageof a biological sample stained with one or more non-nuclear cell markerstains into a segmented image comprising one or more cells havingdelineated nuclei and cytoplasm regions is provided. The computer systemincludes: a processor configured to execute instructions; a memory or astorage device operatively coupled to the processor, one or both of thememory and the storage device configured to store: a model generatedfrom training data comprising a plurality of training images ofbiological samples, the training images comprising regions identified asat least one of nuclei, cells and background; the sample image of thebiological sample stained with the one or more non-nuclear cell markerstains; and processor-executable instructions that, when executed by theprocessor, cause acts to be performed comprising: (a) accessing, in thememory or the storage device, the model and the sample image of thebiological sample stained with the one or more non-nuclear cell markerstains; (b) generating, by applying the model to the sample image, anuclei probability map comprising predicted nuclei regions; and a cellprobability map comprising predicted cell regions; (c) extracting, bythe processor, a binary nuclear mask from the nuclei probability map;(d) extracting, by the processor, a nuclei seeds map from the binarynuclear mask, the nuclei seeds map comprising extracted individualnuclei seeds separated by delineated nuclei regions; (f) applying, bythe processor, the extracted nuclei seeds to the sample image; and (g)transforming the sample image into the segmented image comprising one ormore cells having delineated nuclei and cytoplasm regions.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure will become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a schematic representation of an imaging system according toembodiments of the present disclosure;

FIG. 2 illustrates a block diagram of a computing apparatus according toembodiments of the present disclosure;

FIG. 3 illustrates a deep learning network architecture according toembodiments of the present disclosure;

FIG. 4 illustrates a result of a training or prediction model fornuclei, cells and background labels built according to embodiments ofthe present disclosure;

FIG. 5 illustrates a single channel whole cell segmentation workflowaccording to embodiments of the present disclosure;

FIG. 6 illustrates a single channel whole cell segmentation workflowcorresponding to FIG. 5 with exemplary input and transformed imagesaccording to embodiments of the present disclosure;

FIG. 7 is a block diagram showing steps of transforming a sample imageinto a cell probability map and a nuclei probability map, according toembodiments of the present disclosure;

FIG. 8 is a block diagram showing steps of extracting a binary nuclearmask from a nuclei probability map, according to embodiments of thepresent disclosure;

FIG. 9 is a block diagram showing steps of extracting, from a binarynuclear mask, a nuclei seeds map with individual nuclei seeds separatedby delineated nuclei regions, according to embodiments of the presentdisclosure;

FIG. 10 is a block diagram showing steps of applying extracted nucleiseeds to a sample image, according to embodiments of the presentdisclosure;

FIG. 11 illustrates an algorithm used for computing a cell segmentationquality score, according to embodiments of the present disclosure;

FIGS. 12A-12D illustrate examples of sample images taken at differentmagnifications and stained with various non-nuclear cell markers. InFIG. 12A, sample image is taken at 10× magnification and stained withdsRed. In FIG. 12B, sample image is taken at 10× magnification andstained with TexasRed. In FIG. 12C, sample image is taken at 20×magnification and stained with Cy5. In FIG. 12D, sample image is takenat 20× magnification and stained with dsRed;

FIGS. 13A-13D show segmentation results for an example of Table 2. FIG.13A shows segmentation results using a semi-automated ground truthsegmentation;

FIG. 13B shows segmentation results using a deep learning-based approachaccording to an embodiment of the present disclosure; FIG. 13C and FIG.13D show close-ups of the area in the white box of FIG. 13A and FIG.13B, respectively;

FIGS. 14A-D show segmentation results for another example of Table 2.FIG. 14A shows segmentation results using a semi-automated ground truthsegmentation; FIG. 14B shows segmentation results using a deeplearning-based approach according to an embodiment of the presentdisclosure; FIG. 14C and FIG. 14D show close-ups of the area in thewhite box of FIG. 14A and FIG. 14B, respectively;

FIGS. 15A-15D show segmentation results for yet another example of Table2. FIG. 15A shows segmentation results using a semi-automated groundtruth segmentation; FIG. 15B shows segmentation results using a deeplearning-based approach according to an embodiment of the presentdisclosure; FIG. 15C and FIG. 15D show close-ups of the area in thewhite box of FIG. 15A and FIG. 15B, respectively;

FIG. 16 shows a result of histogram of cell-level quality scores for atotal of 1666 segmented cells from Experiment 1 of Table 2, according toembodiments of the present disclosure;

FIG. 17 shows results of a cross-validation experiment of the deeplearning-based approach, according to embodiments of the presentdisclosure; and

FIGS. 18A-18D illustrate an example of comparison between asemi-automated ground truth segmentation result (FIG. 18A) and a resultof a segmentation using a deep learning-based approach (FIG. 18B)according to embodiments of the present disclosure; FIG. 18C and FIG.18D show close-ups of the area in the white box of FIG. 18A and FIG.18B, respectively.

DETAILED DESCRIPTION

Embodiments of the present disclosure may be performed in situ,including, for example, in intact organ or tissue or in a representativesegment of an organ or tissue. In situ analysis may include cellsobtained from various sources, including an organism, an organ, tissuesample, or a cell culture. Analysis thereof may provide specimen datathat is difficult to obtain should the cells be removed from theirbiological surroundings. Acquiring such may not be possible should thecells within the specimen be disturbed from their natural tissue milieu.

The whole cell segmentation of the present disclosure solved manychallenges faced by the existing segmentation techniques. The benefitsof the system and method of the present disclosure include, but are notlimited to, that it: (a) uses a single channel for segmentation; (b) isapplicable to samples stained with different markers and imaged underdifferent magnifications without the need to customize parameters forindividual samples; (c) provides a whole cell segmentation withclustered cells separated and cell boundaries clearly delineated; andwith comparable accuracy as a ground truth segmentation.

System Overview

The present techniques provide systems and methods for image analysis.In certain embodiments, it is envisaged that the present techniques maybe used in conjunction with previously acquired images, for example,digitally stored images, in retrospective studies. In other embodiments,the images may be acquired from a physical sample. In such embodiments,the present techniques may be used in conjunction with an imageacquisition system. An exemplary imaging system 10 capable of operatingin accordance with the present techniques is depicted in FIG. 1.Generally, the imaging system 10 includes an imager 12 that detectssignals and converts the signals to data that may be processed bydownstream processors. The imager 12 may operate in accordance withvarious physical principles for creating the image data and may includea fluorescent microscope, a bright field microscope, or devices adaptedfor suitable imaging modalities. In general, however, the imager 12creates image data indicative of a biological sample including apopulation of cells 14, shown here as being multiple samples on a tissuemicro array, either in a conventional medium, such as photographic film,or in a digital medium. As used herein, the terms “specimen”,“biological specimen”, “biological material”, or “biological sample”refer to material obtained from, or located in, a biological subject,including biological tissue or fluid obtained from a subject, including,but not limited to, body fluid (e.g., blood, blood plasma, serum, orurine), organs, tissues, biopsies, fractions, and cells isolated from,or located in, any biological system, such as mammals. Specimens,biological specimens, biological samples and/or biological materialsalso may include sections of a biological sample, specimens or materialsincluding tissues (e.g. sectional portions of an organ or tissue) andmay also include extracts from a biological sample, for example, anantigen from a biological fluid (e.g. blood or urine). The specimens,biological specimens, biological samples and/or biological materials maybe imaged as part of a slide.

The imager 12 operates under the control of system control circuitry 16.The system control circuitry 16 may include a wide range of circuits,such as illumination source control circuits, timing circuits, circuitsfor coordinating data acquisition in conjunction with sample movements,circuits for controlling the position of light sources and detectors,and so forth. In the present context, the system control circuitry 16may also include computer-readable memory elements, such as magnetic,electronic, or optical storage media, for storing programs and routinesexecuted by the system control circuitry 16 or by associated componentsof the system 10. The stored programs or routines may include programsor routines for performing all or part of the present techniques.

Image data acquired by the imager 12 may be processed by the imager 12,for a variety of purposes, for example to convert the acquired data orsignal to digital values, and provided to data acquisition circuitry 18.The data acquisition circuitry 18 may perform a wide range of processingfunctions, such as adjustment of digital dynamic ranges, smoothing orsharpening of data, as well as compiling of data streams and files,where desired.

The data acquisition circuitry 18 may also transfer acquired image datato data processing circuitry 20, where additional processing andanalysis may be performed. Thus, the data processing circuitry 20 mayperform substantial analyses of image data, including, but not limitedto, ordering, sharpening, smoothing, feature recognition, and so forth.In addition, the data processing circuitry 20 may receive data for oneor more sample sources (e.g. multiple wells of a multi-well plate). Theprocessed image data may be stored in short or long term storagedevices, such as picture archiving communication systems, which may belocated within or remote from the imaging system 10 and/or reconstructedand displayed for an operator, such as at operator workstation 22.

In addition to displaying the reconstructed image, the operatorworkstation 22 may control the above-described operations and functionsof the imaging system 10, typically via an interface with the systemcontrol circuitry 16. The operator workstation 22 may include one ormore processor-based components, such as general purpose or applicationspecific computers 24. In addition to the processor-based components,the computer 24 may include various memory and/or storage componentsincluding magnetic and optical mass storage devices, internal memory,such as RAM chips. The memory and/or storage components may be used forstoring programs and routines for performing the techniques describedherein that are executed by the operator workstation 22 or by associatedcomponents of the system 10. Alternatively, the programs and routinesmay be stored on a computer accessible storage and/or memory remote fromthe operator workstation 22 but accessible by network and/orcommunication interfaces present on the computer 24. The computer 24 mayalso comprise various input/output (I/O) interfaces, as well as variousnetwork or communication interfaces. The various I/O interfaces mayallow communication with user interface devices, such as a display 26,keyboard 28, mouse 30, and printer 32, that may be used for viewing andinputting configuration information and/or for operating the imagingsystem 10. The various network and communication interfaces may allowconnection to both local and wide area intranets and storage networks aswell as the Internet. The various I/O and communication interfaces mayutilize wires, lines, or suitable wireless interfaces (including WIFI,Bluetooth or cellular telephone interfaces), as appropriate or desired.

More than a single operator workstation 22 may be provided for animaging system 10. For example, an imaging scanner or station mayinclude an operator workstation 22 which permits regulation of theparameters involved in the image data acquisition procedure, whereas adifferent operator workstation 22 may be provided for manipulating,enhancing, and viewing results and reconstructed images. Thus, the imageprocessing, segmenting, and/or enhancement techniques described hereinmay be carried out remotely from the imaging system, as on completelyseparate and independent workstations that access the image data, eitherraw, processed or partially processed and perform the steps andfunctions described herein to improve the image output or to provideadditional types of outputs (e.g., raw data, intensity values, cellprofiles).

Further, it should be understood that the disclosed outputs may also beprovided via the system 10. For example, the system 10 may generatemetrics or values based on the disclosed techniques and may display orprovide other indications of such values via the system 10.

In at least one aspect of the disclosed embodiments, the systems andmethods disclosed herein may be executed by one or more computers orprocessor-based components under the control of one or more programsstored on computer readable medium, such as a non-transitory computerreadable medium. FIG. 2 shows a block diagram of an exemplary computingapparatus 40 that may be used to practice aspects of the presentdisclosure. In at least one exemplary aspect, the system controlcircuitry 16, data acquisition circuitry 18, data processing circuitry20, operator workstation 22 and other disclosed devices, components andsystems may be implemented using an instance or replica of the computingapparatus 40 or may be combined or distributed among any number ofinstances or replicas of computing apparatus 40.

The computing apparatus 40 may include computer readable program code,machine readable executable instructions stored on at least one computerreadable medium 44 or processor-executable instructions (e.g., firmwareor software), which when executed, are configured to carry out andexecute the processes and methods described herein, including all orpart of the embodiments of the present disclosure. The computer readablemedium 44 may be memory device(s) of the computing apparatus 40. Thememory device(s) may include a volatile memory, such as random accessmemory (RAM), and/or a non-volatile memory, such as read-only memory(ROM). The memory device(s) may store a variety of information and maybe used for various purposes. In alternate aspects, the computerreadable program code may be stored in a memory external to, or remotefrom, the apparatus 40. The memory may include magnetic media,semiconductor media, optical media, or any media which may be readableand executable by a computer. Computing apparatus 40 may also includestorage device(s) 46 (e.g., non-volatile storage) such as ROM, flashmemory, a hard drive, or any other suitable optical, magnetic, orsolid-state storage media. The storage device(s) 46 may store data(e.g., input data, processing results, etc.), instructions (e.g.,software or firmware for processing data, etc.), and so forth. Computingapparatus 40 may also include one or more processors 42, to which thememory or other computer readable medium 44 and/or storage device(s) 46is/are operatively coupled, for executing the computer readable programcode stored in the memory or on the at least one computer readablemedium 44. In at least one aspect, computing apparatus 40 may includeone or more input or output devices to allow communication among thecomponents of the exemplary imaging system 10, including, for example,what may be generally referred to as a user interface 48, such as theoperator workstation 22 described above, which may operate the othercomponents included in the imaging system 10 or to provide input oroutput from the computing apparatus 40 to or from other components ofthe imaging system 10.

Development of a Training Model

According to certain embodiments of the present disclosure, a trainingmodel is developed and used in the single channel whole cellsegmentation workflow. The training model may be a deep learning modelwhich may be built on a deep learning framework. The networkarchitecture of the deep learning framework may be a convolutionalneural network that uses a cascade of convolution and deconvolutionlayers to learn a hierarchy of image features (low-level to high-level)that can be used to predict image or pixel labels. Mxnet library and aU-net architecture may be used to compute pixel-level predictions formultiple classes or labels in the deep learning framework.

In certain embodiments, the deep learning model in the whole cellsegmentation may be implemented in python using mxnet, while the nucleiseeds map and the cell segmentations with delineated regions may bedeveloped in python and C++, using ITK. The deep learning model may betrained on an amazon cloud environment (AWS) based on Ubuntu usingnvidia graphics card Tesla K80. The training of the deep learning modelmay take about 11-13 minutes per epoch where epoch is one pass of a fulltraining set, for about 6 hours per cross-validation fold. Applying thetrained model and the post processing steps on a new image may takeabout 4-6 seconds per image.

FIG. 3 illustrates a deep learning network architecture according to anembodiment of the present disclosure in which a U-net architecture isused. A deep learning training model is generated using image patches of160×160 pixels to predict 3 different labels including nuclei, cells andbackground labels. As shown in FIG. 3, from the input image patch of160×160 pixels, a series of 5 convolution and pooling steps are applied.The convolution kernel size is 3×3 and the numbers of filters for the 5layers are 32, 64, 128, 128 and 256, respectively. Thus, the lowestlayer results with 5×5 images.

The training proceeds iteratively, where the number of iterations/epochsis empirically set to a range of about 30-50. In each trainingiteration, the goal is to estimate the network weights such that a lossfunction is minimized More specifically, as shown in Eq. (1) below,l_(n), l_(c) and l_(b) respectively denote the nuclei, cells andbackground labels in a training dataset, and p_(n), p_(c) and p_(n)denote the predictions of the deep learning architecture for the nuclei,cells and background, respectively. Then, a loss function f(x) may bedefined as a root mean square deviation (RSMD) of the prediction and thedenoted label. The loss function may include a constraint for therelationship between the different labels as in Eq. (1):f(x)=w _(n)·RMSD(p _(n) ,l _(n))+w _(c)·RMSD(p _(c) ,l _(c))+w_(b)·RMSD(p _(b) ,l _(b))+w·RMSD(l _(n) +l _(c) +l _(b),1)  Eq (1)where w_(n), w_(c), w_(b) and w represent the weights associated withthe different labels. The weights may be equal to one.

FIG. 4 illustrates a result of a training or prediction model 403 (forexample, a deep learning model) for nuclei, cells and the backgroundlabels built according to embodiments of the present disclosure. Each ofthe three labels has its own predominant characteristics and thecharacteristics are utilized to build the deep learning model. Forexample, a nucleus may have lower intensity signal compared to a cellbody. Generally, the intensity range for the nucleus is close to that ofthe image background. The texture patterns of the brighter cell body(i.e. cytoplasm) may vary from one image to another based on the usedmarker. The training images 401 labeled with labels 402 as illustratedin FIG. 4 may be used as a training set (i.e. training data) forsubsequent image processing.

In certain embodiments, the images used for training may bepreprocessed. By way of example, the image background may be suppressedto correct for uneven illumination. For example, a top-hat filtering maybe applied with a kernel size of 200×200. To account for the differencesin image magnification (and thus pixel size), images may be down-sampledto be approximately at 10× (e.g. pixel Size=0.65×0.65 μm). After that,the input image may be divided into overlapping patches of 176×176pixels, with an overlap of 8 pixels from each side. Therefore, only theinternal 160×160 pixels are unique for each patch. The training data maybe augmented by rotating the original patches by 90 degrees.Additionally and optionally, left-right flipped patches may be includedin the training data. The generated training model may be saved into afile and stored in the memory or the storage device of the computingapparatus 40 to be used in a subsequent step, for example, step 501 or601 as described later in this specification.

Single Channel Whole Cell Segmentation Workflow

FIG. 5 illustrates a single channel whole cell segmentation workflowaccording to embodiments of the present disclosure. FIG. 6 illustrates asingle channel whole cell segmentation workflow corresponding to FIG. 5,with exemplary input and transformed images, according to embodiments ofthe present disclosure.

FIG. 7 illustrates a block diagram showing steps involved in the deeplearning transformation steps illustrated as step 501/601 in FIGS. 5 and6, respectively, for transforming the sample image into a cellprobability map and a nuclei probability map. In step 701, the sampleimage, for example, sample image 51/61 in FIGS. 5 and 6, which may beoptionally preprocessed, is provided to a processor such as theprocessor 42 of computing apparatus 40. In step 702, the sample image isdivided into patches, for example, in an embodiment, the sample image isdivided into 176×176 patches. The sample image may be divided into othersizes of patches suitable for its intended application. In step 703,each pixel of the patches is assigned a predicted label based on anapplied training/prediction model. The label may be selected from agroup of labels including but not limited to, nucleus, cell, andbackground labels. In step 704, the patches comprising the assignedlabels are stitched to form a full predicted image with assignedrespective labels. In step 705, the processor extracts a nucleiprobability map, for example, nuclei probability map 52/62 of FIGS. 5and 6, respectively, and a cell probability map, for example, 53/63 ofFIGS. 5 and 6, respectively, based on the assigned predicted labels.

As illustrated in FIGS. 5 and 6, the nuclei probability map 52/62generated may show the nuclei as brighter regions in the image comparedto other cellular regions, as indicated by the probability scale barsnext to the nuclei probability map 52/62 and the cell probability map53/63. The pixels in the brighter regions in the nuclei probability mapare also referred to as pixels having a higher probability of beingnuclei. Accordingly, nuclei may be defined by these pixels having higherprobability in the nuclei probability map. In a subsequent nucleibinarization transformation step 502/602, a binary nuclear mask 54/64 isextracted from the nuclei probability map. In some embodiments, an imagethresholding of the nuclei probability map may be sufficient to extracta nuclear mask (e.g. a probability higher than 0.5). However, the imagethresholding approach may have drawbacks. First, using imagethresholding may produce false positives due to image artifactsmisclassified as being nuclei. Second, nuclei of adjacent cells may formlarge connected components including multiple nuclei. Therefore, amethod to provide the processor with improved functionality forextracting a binary nuclear mask from the nuclei probability map isenvisioned in the present disclosure.

FIG. 8 illustrates a block diagram showing steps involved in the nucleibinarization transformation steps 502/602 of FIGS. 5 and 6, forextracting a binary nuclear mask from the nuclei probability map.

In step 801, a nuclei probability map 52/62 generated from a prior step501/601 is provided to the processor. The nuclei probability mapcomprises a plurality of pixels with each pixel having an assignedprobability of being a nucleus. The probability of a pixel in the nucleiprobability map is a value selected from a continuous probability scaleof 0-100%. Accordingly, the nuclei probability map may be referred as a“continuous scale nuclei probability map”. As used herein, a “continuousscale” refers to a scale whereby the scale has a minimum and a maximumnumber, and a value on the continuous scale can take any value betweenthe minimum and the maximum numbers. For example, the continuousprobability scale may have a value of 0-100%, and a probability of apixel is assigned a numerical value anywhere along the scale of 0-100%.

In step 802, the processor is configured to identify nuclei withdifferent sizes on the nuclei probability map. In certain embodiments, ablob detection is performed. As used herein, “blob detection” refers toidentification of regions containing blob-like nuclei in an image or adigital object. For example, to detect nuclei with different sizes,regions containing blob-like nuclei in the nuclei probability map areidentified. Performing the blob detection may further include the use ofa Laplacian of Gaussian (LoG) blob detector. In one embodiment, amulti-level (or a multi-scale) Laplacian of Gaussian (LoG) blob detectoris applied at multiple scales to enhance the identified regionscontaining blob-like nuclei. The Laplacian of Gaussian blob detectortakes into consideration the expected morphology as well as theintensity profile of the predicted nuclei. Applying a Laplacian ofGaussian filter at multiple scales improves the detection of nuclei withdifferent sizes. Given a 2-D image I, the Laplacian of Gaussian (LoG) atany pixel (x, y) at scale σ is formulated as shown in Eq. (2):

$\begin{matrix}{{{LoG}\left( {x,y} \right)} = {{- {\frac{1}{{\pi\sigma}^{4}}\left\lbrack {1 - \frac{x^{2} + y^{2}}{2\sigma^{2}}} \right\rbrack}}e^{- \frac{x^{2} + y^{2}}{2\sigma^{2}}}}} & {{Eq}.\mspace{14mu}(2)}\end{matrix}$

The function above is implemented by first convolving the image I by aGaussian filter at scale σ followed by computing the Laplaciandifferential operator on the Gaussian filtered image.

In step 803, the processor applies a multi-level thresholding, forexample, a multi-level Otsu thresholding, to the identified regions ofnuclei with different sizes. The multi-level thresholding may beperformed automatically or with user input. In applying the multi-levelthresholding to the identified regions of nuclei with different sizes,three levels may be assigned, each level defining image background, dimnuclei (blobs) and bright nuclei (blobs), respectively.

In step 804, the processor assigns a binary value to each pixel having aassigned level of the image background, the dim nuclei, and the brightnuclei defined in step 803, respectively. The binary value of each pixelis a value selected from a binary scale. As used herein, a “binaryscale” refers to a scale whereby the scale has only a pair of discretebinary digits (for example, the pair of “0” and “1”, “1” and “2”, “+”and “−”, or the pair of “On” and “Off”, among others), and a binaryvalue can only be selected to be one of the two binary digits on thebinary scale. It is to be appreciated that the examples ofrepresentation of binary digits are given solely for illustrationpurposes and a variety of other pairs of discrete binary digits may beused, the knowledge of which is known to one of ordinary skilled in theart. In some embodiments, the dim and bright nuclei may be combined andassigned one binary value. For example, the background pixels may beassigned a binary value of “0”, and the dim and bright nuclei may becombined and assigned a binary value of “1”.

In step 805, the processor extracts the binary nuclear mask 54/64 fromthe continuous scale nuclei probability map 52/62 based on the binaryvalue assigned to the pixels of the nuclei probability map. Unlike thenuclei probability map in which a pixel has a probability value fallinganywhere within the continuous scale of 0-100%, a pixel in the extractedbinary nuclear mask 54/64 only has a probability value selected to beone of the two binary digits on the binary scale. The extracted binarynuclear mask is therefore also referred to as a “binary scale nuclearmask”. The transformation of the continuous scale nuclei probability map52/62 into a binary scale nuclear mask 54/64 enables the processor tocarry out the image processing in a significantly more efficient way,allowing a much faster image processing. Overall, the current techniquemay process large images in a matter of few seconds. These improvementsare particularly important for applications in which cells being studiedmay undergo rapid and dynamic changes during the course of imaging, forexample, live cell imaging.

Referring back to FIGS. 5 and 6, in a subsequent nuclei separationtransformation step 503/603, the processor further transforms theextracted binary nuclear mask 54/64 into a nuclei seeds map 55/65. Theextracted binary nuclear mask 54/64 obtained from step 502/602 containsnuclei separated from the image background. However, the binary nuclearmask may also contain touching nuclei forming large multi-nucleiconnected components. Using these multi-nuclei connected components asnuclei seeds for cell segmentation in subsequent steps of the workflowwill result in undesirable merging of adjacent cells. Therefore, inaccordance with embodiments of the present disclosure, a nuclei seedsmap 55/65 is further extracted from the binary nuclear mask, the nucleiseeds map having individual nuclei seeds separated by delineated nucleiregions.

FIG. 9 illustrates a block diagram showing the steps involved in thetransformation of a binary nuclear mask into a nuclei seeds map havingindividual nuclei seeds separated by delineated nuclei regions, forexample, step 503/603 of FIGS. 5 and 6. It is to be understood thatwhile a shape-based watershed segmentation approach is shown in FIG. 9for illustration purposes, other approaches may be used, according toembodiments of the present disclosure. In step 901, a distance transformof the binary nuclear mask is determined to generate a distance mapimage in which the value at each pixel in the binary nuclear mask equalsthe pixel's Euclidean distance from the background. Then, in step 902,an extended h-minima transform is applied on the distance map image.This starts by applying a H-minima transform at a level h to suppressall regional minima in the distance map image whose depths are less thanthe value h. In step 903, the processor extracts the regional minima ofthe resulting image with the suppressed regional minima. The parameter hmay be set by a user or a computer and its default value is about 3 μm.In step 904, a seeded watershed transform is applied on the inversedistance transform and uses the regional minima extracted in the step903 as nuclei seeds. The outcome of step 904 is a nuclei seeds maphaving individual nuclei seeds separated by delineated nuclei regions.

Referring back to FIGS. 5 and 6, a cell enhancement transformation step504/604 and a cell delineation transformation step 505/605 illustratethe final steps in the workflow of transforming the sample image intothe segmented image 57/67 having one or more cells with delineatednuclei and cytoplasm regions.

It is to be understood that the cell enhancement transformation step,for example, step 504/604 of FIGS. 5 and 6 is an optional step. In step504/604, the sample image 51/61 is transformed into an enhanced sampleimage 56/66 via a cell enhancement transformation process. The cellprobability map 53/63 generated from the sample image in a deep learningtransformation step 502/602 of FIGS. 5 and 6 is now reapplied back tothe sample image in the cell enhancement transformation step 504/604. Inone embodiment, a pixel-level weighting between the cell probability map53/63 and the sample image 51/61 is performed. For example, theprocessor may use pixel values in the cell probability map as weights toapply the pixel-level weighting to the sample image intensities and togenerate the enhanced sample image. The sample image 51/61, or theenhanced sample image 56/66, is further used by the processor in thecell delineation transformation step, for example, step 505/605 of FIGS.5 and 6.

It is to be noted that using the cell probability map alone is notsufficient to allow the processor to transform the sample image into thesegmented image with one or more cells identified with delineatedregions. To improve the capabilities of the processor and to allow theprocessor to perform the single channel whole cell segmentation on thesample image or the enhanced sample image, the nuclei seeds mapextracted in step 503/603 of FIGS. 5 and 6 is provided to the processor.The processor may extract a nuclei seeds map having individual nucleiseeds in accordance with step 504/604 of FIGS. 5 and 6, and apply theextracted nuclei seeds to the sample image or the enhanced sample imageduring step 505/605 of FIGS. 5 and 6. As a result, the sample image orthe enhanced sample image is transformed into the segmented image havingone or more cells with delineated cellular and/or subcellular regions,including but not limited to nuclei and cytoplasm regions. Asillustrated in an exemplary image 67 in FIG. 6, boundaries of cellsand/or subcellular compartments are delineated. It is to be understoodthat while the boundaries of cells and/or subcellular compartments inthe images are shown in black-and-white line drawings, in certainembodiments, the boundaries may be delineated in different colors torepresent different cell contours.

FIG. 10 is a block diagram showing one embodiment of the transformationsteps invovled in step 504/604 of FIGS. 5 and 6. In an optional step1001, the sample image is preprocessed by the processor for intensityadjustment, for example, by denoising, correcting for backgroundillumination such as non-uniform background illumination, or rescaling(for example, rescaling in log space), or any combination thereof.

In an optional step 1002, the processor uses pixel values in the cellprobability map as weights to apply a pixel-level weighting to thesample image intensities and to transform the sample image into anenhanced sample image.

In step 1003, the processor determines the image background and assignsbackground labels based on an expected area of all cells. The processorapplies the extracted nuclei seeds to the sample image or the enhancedsample image. The expected area of the cells may be determined based onnumbers of extracted nuclei seeds in the nuclei seeds map and an averageexpected area for each cell, based on the assumption that one nucleiseed corresponds to one cell. A multi-level thresholding, for example, amulti-level Otsu thresholding, may also be applied to the image. Anoptimal Otsu threshold is selected to correspond to an area estimated tobe the closest to the expected cell area. The regions that do notcorrespond to the expected cell area are assigned background labels.

In step 1004, the assigned background labels from step 1003, along withthe extracted nuclei seeds map comprising individual nuclei seedsseparated by delineated nuclei regions, are used in a seeded watershedsegmentation of the image. This approach allows for the identificationand separation of cells. For each nucleus in the image, the approachwill identify a corresponding cell.

The single channel whole cell segmentation workflow in accordance withembodiments of the present disclosure provides the processor withimproved capabilities not readily obtainable with existing techniques,for example, the image processing ability of the processor to transforma sample image of a biological sample stained with one or morenon-nuclear cell marker stains into a segmented image having one or moreindividual cells identified with delineated nuclei and cytoplasmregions. In addition, the techniques of the present disclosure havesuccessfully solved multiple technical challenges faced by the existingtechniques. For example, the techniques of the present disclosureprovide the processor with the ability to carry out a single channelwhole cell segmentation using a single channel of a microscope. This isin contrast to segmentation techniques using two or more channels inwhich one of the channels is reserved to be used with a nuclei-specificcell marker stain.

According to embodiments of the present disclosure, the sample isstained with only non-nuclear cell marker stain(s) which do notspecifically stain cell nucleus, thus eliminating the need for thereserved channel for the nuclei-specific cell marker stain, andmaximizing the use of limited numbers of channels of most instruments.Furthermore, by eliminating the need of using a nuclei-specific cellmarker stain, the present techniques provide significant technologyimprovements in various technical fields including, but not limited to,live cell imaging in which a potentially detrimental toxic effect of anuclear cell marker stain to the cells, especially the cells undergoinglive cell imaging, can now be avoided.

As noted herein and used throughout in the present disclosure, the term“nuclei-specific cell marker stain” may also be referred to as a“nuclear cell marker stain”, a “nuclear marker stain”, or a “nuclearstain”, which may specifically stain a cell nucleus. In someembodiments, the nuclear cell marker stain is a DAPI stain. In someembodiments, the nuclear cell marker stain may include a nuclei-specificdye or stain such as a fluorescent dye specific for nuclear DNA. In someembodiments, the nuclear cell marker stain may include a nuclear cellmarker antibody that detects a cell marker protein specific to thenucleus of a cell and can aid in the study of the morphology anddynamics of the nucleus and its structures. The cell marker proteinspecific to the nucleus may include, but is not limited to, ASH2L,ATP2A2, CALR, CD3EAP, CENPA, COL1A1, DDIT3, EIF6, ESR1, ESR2, GAPDH,H2AFX, H2AFZ, HIF1A, HIST1H4A, Ki67, LaminA/C, MYC, NOP2, NOS2, NR3C1,NUP98, pCREB, SIRT1, SRSF2, or TNF, or any combination thereof.

As noted herein and used throughout in the present disclosure, the term“non-nuclear cell marker stain” may also be referred to as a“non-nuclear marker stain”, or a “non-nuclear stain”. In someembodiments in accordance with the present disclosure, the nuclear cellmarker stain is a DAPI stain and the non-nuclear cell marker stain is astain that is not DAPI (i.e. a non-DAPI stain). In some embodiments, thenon-nuclear cell marker stain may include a dye or stain such as afluorescent dye that is not specific for nuclear DNA. In someembodiments, the non-nuclear cell marker stain may also include anon-nuclear cell marker antibody that detects a non-nuclear cell marker,for example, a non-nuclear cell marker protein that is not specific tothe nucleus of a cell. In some embodiments, the non-nuclear cellmarker(s) include structural markers such as any plasma membrane markerincluding, but not limited to, NaKATPase, PMCA, pan-Cadherin, CD98,CD45, ATP2A2, C3, CD40L, CD27, CD40, ESR1, CD95, DIg4, Grb2, FADD,GAPDH, LCK, MAPT, IL6, Membrin, NOS3, RYR1, P4HB, RAIDD, CALR, etc., andany cytoplasmic marker including, but not limited to, ACTB, AHR, CALR,DDIT3, DLg4, ESR1, GAPDH, HIF1A, HSPA1A, NOS2, NOS3T, NR3C1, MAPT, RYR1,etc. Other non-limiting examples of non-nuclear cell marker(s) include,but are not limited to, ACTC1, ACTC2, HSPB1, KRT17, MAP1LC3B, NFATC2,TNNT2, TUBA1A, TUBB3, FP3, S6, pS6, CFP4, Glu1, CFP5, pS6235, CFP6,CFP7, FOXO3a, CFPS, pAkt, CFP9, pGSK3beta, pan-Keratin, etc., and anyother non-nuclear cell marker specific for a certain application that itis intended for. It is to be understood that a combination ofnon-nuclear cell marker stains may be used in the same channel toprovide a uniform stain if one individual non-nuclear cell marker staindoes not stain the whole cellular compartments. It is also to beunderstood that a wide variety of non-nuclear cell marker stains(s) areknown to one of ordinary skilled in the art and are intended to bewithin the scope of the non-nuclear cell marker stain(s) in accordancewith the present disclosure.

Additional benefits provided by the present disclosure include, forexample, that the techniques are applicable to various cell types andsamples. Furthermore, as shown in the Examples section below and inaccordance with embodiments of the present disclosure, samples stainedwith a wide variety of non-nuclei cell marker stains and imaged withvarious magnifications may be subjected to the same whole cellsegmentation workflow without requiring additional modifications to theworkflow. The elimination of customized modifications enables theprocessor to perform the image processing in a much faster and moreefficient manner, which is important for applications such as ones inwhich cells undergo rapid and dynamic changes during the course ofimaging.

Assessment of Segmentation Results

In certain embodiments, an assessment of segmentation results may beperformed. The computing apparatus 40 includes an assessment unit andthe assessment unit is configured to perform a quality control (QC)assessment and provides the QC assessment results in a form of a qualityscore. The QC assessment may be based on a metric, for example, asimilarity metric. The similarity metric may include, for example, acell segmentation similarity metric defined as a metric to compare acell segmentation result to a gold standard (or a ground truth) cellsegmentation result. The ground truth cell segmentation may be anautomatic or a semi-automatic ground truth cell segmentation asdiscussed in the Examples section below. In some embodiments, the cellsegmentation similarity metric is applied to a segmented image, toassess the quality or goodness of the cell segmentation, and to generatea cell segmentation quality score. In some embodiments, where thesegmented image contains one or a few segmented objects, a binarysimilarity metric may be used (e.g. Dice overlap ratio metric). Incertain embodiments, where the segmented image contains a large numberof segmented objects (e.g. hundreds or thousands of segmented cells), abinary similarity metric may not be sufficient, and a cell segmentationsimilarity metric defined by the algorithm in FIG. 11 may be used for QCassessment.

EXAMPLES

Dataset

Table 2 below shows details of datasets and experimental conditions usedin the Examples section. The datasets were used for either training amodel in accordance with aspects of the present disclosure or fortesting the segmentation method in accordance with embodiments of thepresent disclosure.

Number Cytoplasm Training or Testing of marker Pixel Size Plate # DataSet Experiment 1 Experiment 2 Experiment 3 images channel Image Size(μm) 1 FYVE Helo, Training Training & Training 22 Green-dsRed 2048 ×2048 0.325 × 0.325 #1 Testing Red-Cy5 2 Lysosome Training Training &Training 12 Green-dsRed 2048 × 2048 0.650 × 0.650 assay Testing(Fibroblast) 3 INCA6K Training Training & Training 24 Red-Cy5 2048 ×2048 0.325 × 0.325 Testing 4 Mito 1 Training Training & Training 30TexasRed- 2048 × 2048 0.325 × 0.325 Testing TexasRed 5 FYVE Helo, 10Training Training & Training 20 Green-dsRed 2048 × 2048 0.325 × 0.325 #2& 10 Testing Red-Cy5 Testing 6 Cell Health Not Used Testing 15 TexasRed-2048 × 2048 0.325 × 0.325 CH (Mito5) TexasRed

In Table 2, a dataset including data from six plates (Plates #1-6) wasused, each plate containing images of samples subjected to a pluralityof assays in a 96-well microplate (referred to as “plates” forsimplicity). Samples were stained with different fluorescent dyes asnon-nuclear cell marker stains to identify cell body/cytoplasm. In Table2, fluorescent dyes such as dsRed, Cy5, TexasRed were used. However, inother applications, non-nuclear cell marker stains suitable for thoseparticular applications may be used. The choice of a suitable cellmarker stain according to its intended application is within theknowledge of one of ordinary skill in the art. The images were acquiredusing GE's INCell Analyzer systems. The plates were scanned at variousmagnifications including 10× (pixel size 0.65×0.65) and 20× (pixel size0.325×0.325). Other magnifications may be used, including, but notlimited to, 40× and 60× (data not shown). Regardless of themagnifications, each image was 2048×2048 pixels in size. Only a smallsubset of the wells in each plate (e.g. one or two rows of each plate)were used in the experiments. Experiments 1-3 of Table 2 each had atotal of 123 images. It is to be understood that other numbers of imagesmay be used. The selection of the numbers of images and choice ofdataset according to its intended application is within the knowledge ofone of ordinary skill in the art. FIGS. 12A-12D illustrates example ofsample images stained with various non-nuclear cell marker stains andimaged at different magnifications: (A) 10×, dsRed channel; (B) 10×,TexasRed channel; (C) 20×, Cy5 channel; and (D) 20×, dsRed channel.

In order to generate a training model, for example, a deep-learningbased model, a set of segmented images, referred to as ground truthsegmentation is used. The process of generating a training model may bereferred to as “train a model”. The ground truth segmentation may beused to train the model as well as to evaluate the results ofsegmentation using the training model. Ideally, a human expert wouldcreate such ground truth segmentation. However, this is verytime-consuming since the images may contain several hundreds of cells.To overcome this limitation, the model according to aspects of thepresent disclosure is trained using automatically obtained sub-optimalsegmentations (referred to as “automated ground truth segmentation”). Insome embodiments, sample images stained with both a nuclei-specificmarker stain and a non-nuclear cell marker stain are subjected to atwo-channel cell segmentation technique previously developed, togenerate a two-channel segmentation to be used as an automated groundtruth segmentation in training the model.

In certain embodiments, in addition to the automatically generatedground truth segmentations, images may be semi-automatically segmentedby an expert (referred to as “semi-automated ground truth segmentation”)and used to validate automated segmentation results. For semi-automatedground truth segmentation, an expert may use a software (for example,CellProfiler, an open-source software) to generate an initialsub-optimal segmentation. Then, the expert may use the software tomanually refine/edit the segmentation results by splitting, merging,adding and removing cells.

Table 2 summarizes three experiments (Experiment 1-3). In all threeexperiments, the automated two-channel ground truth segmentations wereused for training a deep learning model. The dataset may be divideddifferently into a dataset used for training (i.e. a training set), adataset used for testing (i.e. a testing set), and a dataset used forvalidating the segmentation results for each respective experiment.

Experiment 1: Segmentation

In Experiment 1, the plates #1-5 in Table 2 were used, yielding a totalof 108 sample images. Among the 108 images, 10 images were used as atesting set, and the remaining 98 images were further divided into atraining set (88 images) and a validation set (10 images). Thesegmentation results of the 10 images of the testing set were evaluatedby comparing them to the semi-automated ground truth segmentations.FIGS. 13A-13D show the segmentation results for one sample of Experiment1 of Table 2. FIG. 13A (top, left column) illustrates the segmentationresult of an image subjected to the semi-automated ground truthsegmentation. FIG. 13B (top, right column) illustrates the segmentationresult of an image subjected to segmentation technique according to anembodiment of the present disclosure. FIGS. 13C and 13D (bottom row)show close-ups of the area in the white box of FIGS. 13A and 13B,respectively. High similarity was observed for the two segmentationresults.

FIGS. 14A-14D show the segmentation results for another example ofExperiment 1 of Table 2. FIG. 14A (top, left column) illustrates thesegmentation result of an image subjected to the semi-automated groundtruth segmentation. FIG. 14B (top, right column) illustrates thesegmentation result of an image subjected to segmentation according toan embodiment of the present disclosure. FIGS. 14C and 14D (bottom row)show close-ups of the area in the white box of FIGS. 14A and 14B,respectively.

FIGS. 15A-15D show the segmentation results for yet another example ofExperiment 1 of Table 2. FIG. 15A (top, left column) illustrates thesegmentation result of an image subjected to the semi-automated groundtruth segmentation. FIG. 15B (top, right column) illustrates thesegmentation result of an image subjected to segmentation according toan embodiment of the present disclosure. FIGS. 15C and 15D (bottom row)show close-ups of the area in the white box of FIGS. 15A and 15B,respectively.

To further assess the quality of the cell segmentation results, a cellsegmentation similarity metric may be applied to segmented images toquantitatively compare single channel whole cell segmentation resultsaccording to embodiments of the present disclosure with ground truthsegmentation results, including automated ground truth segmentationand/or semi-automated ground truth segmentation results. The cellsegmentation similarity metric may be defined by the algorithm of FIG.11. In Experiment 1, 10 test images were subjected to a semi-automatedground truth segmentation. A total of 1666 cells in all the images weresubjected to the single channel whole cell segmentation in accordancewith embodiments of the present disclosure. A segmentation quality scorewas then determined for each individual cell of the 1666 cells bycomparing the segmentation result of the individual cell to acorresponding ground truth segmentation of the same cell.

FIG. 16 shows a histogram of the cell-level segmentation quality scoresfor cells in the samples images of Experiment 1. The overall (average)cell-level segmentation quality score was found to be about 0.87. Theoverall cell-level scores by comparing the segmentation results to thetwo-channel segmentation results resulted in an average score of about0.86. For comparison, comparing the two-channel segmentation results tothe semi-automated ground truth segmentation resulted in an averagescore of about 0.93.

In addition to the cell-level segmentation quality scores, animage-level quality score was also determined by averaging at the imagelevel according to the workflow described in FIG. 11. The results of theimage-level segmentation quality assessment are shown in Table 3.

TABLE 3 Image Level Segmentation Comparisons for the first experimentDeep Learning to Deep Learning to Two-Channel to Two-Channel GroundTruth Ground Truth Image ID Similarity Similarity Similarity 1 0.88 0.900.94 2 0.86 0.85 0.94 3 0.89 0.91 0.94 4 0.92 0.91 0.91 5 0.88 0.90 0.936 0.83 0.84 0.94 7 0.76 0.80 0.87 8 0.72 0.72 0.96 9 0.83 0.86 0.94 10 0.89 0.90 0.95 Average 0.85 0.86 0.93

Table 3 shows the segmentation results of samples in Experiment 1, inaccordance with embodiments of the present disclosure. As shown in Table3, 10 images (Image ID #1-10) were selected and three segmentationcomparisons were performed. The first segmentation result set wasdeveloped by semi-automated segmentation performed by an expert in whichthe expert used a software to manually edit or modify the segmentation.This was referred to as “ground truth segmentation” in Table 3. Thesecond segmentation result set was generated automatically by applying atwo-channel segmentation technique previously developed and was referredto as “two channel segmentation” in Table 3. The third segmentationresult set was developed by the single channel whole cell segmentationtechnique according to embodiments of the present disclosure and wasreferred to as “deep learning segmentation” in Table 3. The results ofdeep learning segmentation were compared to the two-channel segmentation(left column “deep learning to two-channel similarity”) as well as tothe semi-automated ground truth segmentation (middle column “deeplearning to ground truth similarity”), with an average image-levelquality score of 0.85 and 0.86, respectively. For comparison, results ofthe two-channel segmentation, which were used to train a model used inthe single channel whole cell segmentation, are compared to thesemi-automated ground truth segmentation results (right column in Table3, “two-channel to ground truth similarity”). The comparison resulted inan average image-level score of 0.93.

Experiment 2: Cross Validation

FIG. 17 shows the results of a cross validation experiment—Experiment 2in Table 2. In this experiment, a 10-fold cross validation was performedto build a receiver operating curve (ROC). The cross validation was thenused to assess the Area under the curve (AUC) and the accuracy (ACC) ofthe nuclei probability map and the cell probability map. Thecross-validation used 108 independent sample images taken from the first5 plates #1-5 in Table 2 (Experiment 2). In each cross-validationiteration, the images were split into three non-overlapping sets ofimages: a training set (88 images), a validation set (10 images) and atest set (10 images).

Table 4 shows the results of Area Under Curve (“AUC”) and Accuracy(“ACC”) of the Receiver Operating Curve of FIG. 17. AUC for both nucleiand cell are larger than 0.95, and ACC are about 0.915 and 0.878 for thenuclei and cell, respectively. The ACC values are determined using athreshold of 0.50. Note that the AUC and ACC are determined on thebinary nuclear masks, assuming that all nuclei are assigned one binaryvalue and the reminders of all cells are assigned the other binaryvalue.

TABLE 4 Area Under the Receiver Operating Curve (AUC) and Accuracy. Mean± Standard deviation (range) Nuclei Cell AUC 0.967 ± 0.004 (0.962,0.971) 0.952 ± 0.028 (0.886, 0.976) Accuracy 0.915 ± 0.009 (0.901,0.931) 0.878 ± 0.031 (0.810, 0.917)

Table 5 shows a summary of the segmentation accuracies for the differentdatasets (plates). The overall accuracy for the four datasets wascomputed to be about 0.84.

TABLE 5 Summary of segmentation accuracy for the 10-fold crossvalidation experiment Number of Segmentation Data Set Detected CellsAccuracy FYVE Hela, #1&2 6378 0.86 ± 0.14 Lysosome assay 2162 0.62 ±0.09 (Fibroblast) INCA6K 2735 0.91 ± 0.12 Mito 1 7961 0.85 ± 0.17 Total# Cells = Avg. Accuracy = 19236 0.84 ± 0.14

Experiment 3: Model or Network Training

A total of 108 images from the plates #1-5 of Table 2 were used to trainthe model/network, for example, a deep learning model. The 108 images ofExperiment 3 were divided into 98 images as a training set and 10 imagesas a validation set. Then, the model/network was tested using 15 imagesfrom the plate #6 of Table 2 as a testing set. Since no semi-automatedground truth segmentation was available for the testing dataset,two-channel segmentations were generated and used as ground truthsegmentations after a qualitative assessment by an expert. Applying theprocess as illustrated in Experiment 1 discussed in the earlier section,an overall cell-level segmentation quality score was determined to beabout 0.84. Then, the image-level segmentation quality scores weredetermined by averaging the cell-level quality scores for each image,using the similarity metric as defined in the algorithm as illustratedin FIG. 11.

Table 6 shows the results of image-level segmentation quality scores forthe 15 images from the sixth plate. Most of the image-level segmentationquality scores range between about 0.8 and about 0.9, and the averageimage-level segmentation quality score is about 0.84, which is similarto the overall cell-level segmentation quality score for the same plate#6. The results were reviewed and approved by an expert.

TABLE 6 Image Level Segmentation Comparisons for the third experimentDeep Learning to Two-Channel Image ID Similarity 1 0.88 2 0.83 3 0.82 40.81 5 0.84 6 0.76 7 0.85 8 0.87 9 0.84 10 0.80 11 0.86 12 0.85 13 0.8514 0.88 15 0.79 Average 0.84

FIGS. 18A-18D illustrate a comparison of a semi-automated ground truthsegmentation for an example of Experiment 3 of Table 2 to the whole cellsegmentation in accordance with aspects of the present disclosure. FIG.18A (top, left column) illustrates the segmentation result of an imagesubjected to the semi-automated ground truth segmentation. FIG. 18B(top, right column) illustrates the segmentation result of an imagesubjected to segmentation according to an embodiment of the presentdisclosure. FIGS. 18C and 18D (bottom row) show close-ups of the area inthe white box of FIGS. 18A and 18B, respectively.

Both the qualitative and quantitative results of the Example sectiondemonstrate that the segmentation technique in accordance withembodiments of the present disclosure provides comparable accuracy as aground truth segmentation, with the additional benefits of transforminga sample image of a biological sample stained with one or morenon-nuclear cell marker stains into a segmented image comprising one ormore cells having delineated nuclei and cytoplasm regions, utilizing asingle channel to achieve the segmentation. Thus, the techniquesillustrated in the present disclosure clearly provide improved technicalsolutions and solve challenges faced by existing techniques in thefields including but not limited to, studying cells.

This written description uses examples as part of the disclosure,including the best mode, and also to enable any person skilled in theart to practice the disclosed implementations, including making andusing any devices or systems and performing any incorporated methods.The patentable scope is defined by the claims, and may include otherexamples that occur to those skilled in the art. Such other examples areintended to be within the scope of the claims if they have structuralelements that do not differ from the literal language of the claims, orif they include equivalent structural elements with insubstantialdifferences from the literal languages of the claims.

The invention claimed is:
 1. A computer-implemented method fortransforming a sample image of a biological sample stained with only oneor more non-nuclear cell marker stains which do not specifically staincell nuclei into a segmented image comprising one or more cells havingdelineated nuclei and cytoplasm regions, the method comprising: (a)providing a computer system having: a processor configured to executeinstructions; and a memory or a storage device operatively coupled tothe processor, wherein at least one of the memory and the storage deviceis configured to store: a model generated from training data comprisinga plurality of training images of biological samples, the trainingimages comprising regions identified as at least one of nuclei, cellsand background; the sample image of the biological sample stained withthe one or more non-nuclear cell marker stains; and processor-executableinstructions, (b) accessing, in the memory or the storage device, themodel and the sample image of the biological sample stained with the oneor more non-nuclear cell marker stains; (c) generating, by applying themodel to the sample image of the biological sample stained with the oneor more non-nuclear cell marker stains, a nuclei probability mapcomprising predicted nuclei regions; and a cell probability mapcomprising predicted cell regions; (d) extracting, by the processor, abinary nuclear mask from the nuclei probability map; (e) extracting, bythe processor, a nuclei seeds map from the binary nuclear mask, thenuclei seeds map comprising extracted individual nuclei seeds separatedby delineated nuclei regions; (f) applying, by the processor, theextracted nuclei seeds to the sample image; and (g) transforming thesample image into the segmented image comprising one or more cellshaving delineated nuclei and cytoplasm regions.
 2. A method according toclaim 1, wherein the sample image comprises a plurality of pixels andstep (c) further comprises determining a probability scale of each pixelof the plurality of pixels being a background pixel, a cell pixel or anucleus pixel.
 3. A method according to claim 2, wherein the probabilityscale is a continuous scale.
 4. A method according to claim 1, whereinstep (d) further comprises performing at least one of a blob detectionand a multi-level thresholding.
 5. A method according to claim 4,further comprising: identifying regions of nuclei with different sizeson the nuclei probability map; applying a multi-level thresholding onthe identified regions of nuclei; assigning binary value to each pixelof a plurality of pixels in the nuclei probability map; and extractingthe binary nuclear mask from the nuclei probability map.
 6. A methodaccording to claim 1, wherein step (e) further comprises performing ashape-based watershed segmentation.
 7. A method according to claim 6,further comprising: determining, by the processor, an inverse distancetransform of the binary nuclear mask; applying, by the processor, anextended h-minima transform to the determined inverse distancetransform; extracting, by the processor, regional minima from the binarynuclear mask; and applying, by the processor, using the regional minimaas seeds, a seeded watershed transform to the determined inversedistance transform.
 8. A method according to claim 7, wherein applyingthe extended h-minima transform comprises applying an H-minima transformat a depth h to suppress regional minima having a depth less than h, andextracting the regional minima from a resulting image with thesuppressed regional minima.
 9. A method according to claim 8, whereinthe depth h is a user-defined value.
 10. A method according to claim 8,wherein the depth h has a default value of 3 μm.
 11. A method accordingto claim 1, wherein the sample image is preprocessed for intensityadjustment, the preprocessing comprises performing, by the processor, atleast one of the steps of: denoising of the sample image; backgroundillumination correction of the sample image; and rescaling of the sampleimage.
 12. A method according to claim 1, wherein the sample image is anenhanced sample image generated by applying a pixel-level weighting tothe sample image.
 13. A method according to claim 1, wherein step (g)further comprises determining the image background and assigningbackground labels based on multi-level thresholding.
 14. A methodaccording to claim 13, wherein the multi-level thresholding is performedby: identifying, by the processor, a number of nuclei regions in thenuclei probability map and an expected area of cell regions in the cellprobability map, and selecting, by the processor, a threshold valuewhich results in an area estimated to be closest to the expected area ofthe cell regions.
 15. A method according to claim 12, wherein step (g)further comprises performing a seeded watershed segmentation using theassigned background labels and extracted nuclei seeds comprisingindividual nuclei seeds separated by delineated nuclei regions.
 16. Acomputer system for transforming a sample image of a biological samplestained with only one or more non-nuclear cell marker stains which donot specifically stain cell nuclei into a segmented image comprising oneor more cells having delineated nuclei and cytoplasm regions, thecomputer system comprising: a processor configured to executeinstructions; a memory or a storage device operatively coupled to theprocessor, one or both of the memory and the storage device configuredto store: a model generated from training data comprising a plurality oftraining images of biological samples, the training images comprisingregions identified as at least one of nuclei, cells and background; thesample image of the biological sample stained with the one or morenon-nuclear cell marker stains; and processor-executable instructionsthat, when executed by the processor, cause acts to be performedcomprising: (a) accessing, in the memory or the storage device, themodel and the sample image of the biological sample stained with the oneor more non-nuclear cell marker stains; (b) generating, by applying themodel to the sample image of the biological sample stained with the oneor more non-nuclear cell marker stains, a nuclei probability mapcomprising predicted nuclei regions; and a cell probability mapcomprising predicted cell regions; (c) extracting, by the processor, abinary nuclear mask from the nuclei probability map; (d) extracting, bythe processor, a nuclei seeds map from the binary nuclear mask, thenuclei seeds map comprising extracted individual nuclei seeds separatedby delineated nuclei regions; (f) applying, by the processor, theextracted nuclei seeds to the sample image; and (g) transforming thesample image into the segmented image comprising one or more cellshaving delineated nuclei and cytoplasm regions.